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Foreword 



The Strategies for Preschool Intervention in Everyday Settings (SPIES) is an 
attempt to orient parents and teachers of preschool children to ways of 
dealing with children with disabilities. Based upon a commitment to early 
intervention, it was a project funded by the State of Utah in conjunction with 
the Center for Persons with Disabilities (CPD) at Utah State University. The 
primary vehicle for providing this training is a series of six videotapes, with 
an accompanying printed manual. The producer of this video series was Mr. 
Tom Risk, Director of K-S AR Productions, with help from K-S AR staff 
members and from faculty members of the Dept, of Special Education at 
Utah State University. The entire project was under the supervision of Dr. 
Sarah Rule, Associate Director of the Center for Persons with Disabilities. 

As part of an ongoing arrangement with this office, members of the 
Introduction to Research for the Classroom Teacher course (Education 6550) 
used this as a class project during Spring Semester, 1999. Class members 
worked in teams to use the class-designed instruments to interview experts 
and potential learners about their reaction to these materials. The model for 
evaluation used was Martin Tessmer’s Layers of Necessity model (from 
Pl annin g and Conducting Formative Evaluations, 1997, London: Kogan 
Page). This is our written evaluation report. 

While there were three primary authors, other class members contributed as 
well. Their contribution should be acknowledged. Other class members 
were: 



Brian Anderson 
Marla Bailey 
Cherie Doe 
Jeramy Jenkins 
Jennifer Merritt 
Judy Ricks 
Dan Trimble 



Mindy Anderson 
Deaima Campbell 
Lory Farr 
Dixie Jenson 
Stephanie Podgorski 
Chloe Russell 
Wendy Warner 



We express our appreciation to all the participants who made this 
evaluation possible. 



The SPIES Research Project: 

An Evaluation of the SPIES Video Series 



Written by Ginger Utley and April Qian 
Research conducted by the class members of 
Dr. Nick Eastmond’s Educational Research Course 6550 
Utah State University, 1999 



Introduction: 

This report details the methods and results of the SPIES research project that was conducted 
under the direction of Dr. Nick Eastmond and completed by his Education 6550 class members. 
The client for the study is Tom Risk, producer of the SPIES video series, who employed Dr. 
Eastmond’s class to evaluate the video series based on the specific criteria of written, audio and 
visual components. 

Why Was the Study Conducted? 

This study was conducted to evaluate the SPIES video series. The evaluation was based on the 
criteria of appeal, effectiveness and user-friendliness of the written, audio and video materials, in 
both the English and Spanish versions. Respondents included experts in technical fields, special 
education, and parents of disabled children. The results of the study follow and incorporate 
numeric averages of responses as well as open-ended comments made by survey respondents. 



Project Goals: 

Obtain wide variety of user responses 

Obtain information on written, audio, and 
video quality 

Obtain information for future projects (how to 
improve similar video series) 



Research Methods: 

Survey experts and users who will provide 
valuable, varying responses 

Ask a variety of questions based on a scale 
that can be averaged and rated 

Ask open-ended questions regarding 
suggestions and ideas for improvements 



What Did We Expect to Find in This Study? 

We expected to get similar responses from each of the categories of respondents who were 
interviewed: technology experts, special education teachers and parents of disabled children. We 
hoped that there would be similarities within group respondents and slight variations between 
each respondent population based on expertise and experience. The variations were expected to 
form a valuable portion of the results in that they would provide three distinct, important 
perspectives from which to draw conclusions. 
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Who Participated in this Study? 



The subjects were selected by each research team. Many of the technical and educational experts 
were on-campus faculty members, and parent-respondents were selected from among each 
team’s personal connections and/or referrals. 

How Was the Research Conducted and Compiled? 

The researchers included two students per team with a total of six teams. Most teams opted to 
separate respondent groups for interviewing purposes: one person interviewed technical experts 
while the other person interviewed teachers and parents. The teams then came together and 
compiled their results in a team report. We then compiled all the team’s results into a master list 
that generated totals and averages of responses. 

Design of the Questionnaire: 

The questionnaire was designed to include responses that could be measured and averaged as 
well as open-ended questions for original thoughts and ideas. See Appendix B for a model 
questionnaire; each survey team modified the questionnaire as they felt necessary. 

Profile of Respondents: 

We felt it necessary to gather background information on respondents in order to correctly 
categorized the data we received. The major characteristics of the respondents follows and is 
based on three criteria; type of expertise, background, researcher confidence in responses. 
Overall, interviewers had a relatively high level of confidence in their respondents. 



Video Number 


Field of Expertise 


Background Info 


Level of Confidence 


1 


Special Ed 


Tutor 


High 




Speech-language 

pathology 


Special ed student 


Good 




Communication 

disorders 




High 


1 

(Spanish) 


Speech-language 

pathology 


Parents speak 
Spanish 


Good 




Audiology, 
education, Spanish 




Good 


2 


Special ed teacher 


Special ed 
specialization 


High 
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Video Number 


Field of Expertise 


Background Info 


Level of Confidence 




Childhood education 
associate professor 


Parent & school 
teacher 


High 




Elementary education 
associate professor 


Parent and special ed. 
aide 


High 


3 


Early childhood 
special ed (2) 


Parents with back- 
ground in instruc- 
tional design 


High 




Instructional video 
business (2) 


Design 


High 


4 


Educational video 
director 


Parent with special 
ed experience 


Reliable 




Experienced teacher 
specialized in 
teaching and video 
making 


Parent with special 
ed and comm, 
disorders experience 






Family & human 
development, adult 
behavior problems 
program 








Middle school 
special ed teacher 






5 


Experienced first 
grade teacher 


Parent w/background 
in comm disorders 


High 




Preschool teacher 


Parent 


Medium 


6 


Audiology, high 
fidelity 


Family and Human 
Dev. Major 


High 



Among the experts, 5 (25%) are audio-visual production experts, 1 1 (55%) are educators of 
various kinds, 4 (20%) are speech or communication specialists, and learners are either teachers 
or parents. 



Findings/Results of the Study: 



The first part of the questionnaire was focused on determining the quality of the instmctional 
material. Questions were asked about the Appeal , Effectiveness and User-friendliness of the 
written, audio and video materials. The data has been gathered and reported in a mamier that 
applies to the videos as a series; for a quality evaluation of each individual video-tape, please 
refer to individual team reports and questionnaires. To ensure the raw material is consistently 
interpreted, a record of all the original data is attached as Appendix A. 

Overall Quality of the Videos: 

For the 22 questions dealing with overall quality of the videos, and based on the 1-5 scale 
discussed previously, the average response was 3.90-a rating of “good” most closely matches the 
numeric average. 

The questions that earned the highest scores from respondents are the following: 

• Up-to-date information 

• Transparent camera work 

• Use of objectives 

• Cohesive content (consistent to main topic) 

• Use of understandable, up-to-date language 

Learner Responses vs. Expert Responses: 

When we break down the results by expert and learner, we found slightly different results. 

We consider a standard deviation of 0.1 and above a significant difference. 

In regards to the high scores, on several occasions, experts responded with higher scores than did 
the learners. For example Questions 6, 8, 9,10, 11 & 13 received higher scores from experts 
while Questions 1 , 2, 5, 12, andl4 earned higher scores from learners. This is a phenomenon we 
expected to find, and we feel it adds validity to the study. 
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Report of 1-5 Scale Findings: 

The following table incorporates questionnaires responses; the key ideas and opinions shown 
here represent major points taken from the six group reports. The numbers in parentheses 
represent the most common responses from each group report: on Appeal of Written Material, for 
instance, tliree teams’ reported the most frequent response as “Good”. The other comments are 
opinions and suggestions offered from various respondents; we incorporated those that occurred 
more than once. 





Written Material 


Audio 


Video 


Appeal 


Good (3) 

Nice outline 
Good supplement 


Good (3) 

Fair (2) 

Sufficient 

Narration uninteresting 
Need Spanish audio 


Good (4) 

Good sequencing 
Use teachers more in 
teaching concepts 


Effectiveness 


Good (2) 
Readable 
Understandable 
Fair 


Good (4) 

Fair 
Audible 
Dry sometimes 
Subtitles difficult to 
follow 


Good (4) 

Common sense info 
Real life examples 
Good content 
Helpful ideas 


User- 

Friendliness 


Good (3) 

Easy to follow 


Good 
Fair (2) 


Good (3) 

Easy to follow 
Fair 

Good translation 
Who was video made 
for? 



ERIC 



8 



Report of Open-Ended Questions Findings (including comments/suggestion): 

The following results are taken from the open ended questions asked to each respondent. 
The comments, suggestions, and ideas that occurred more than once have numbers of 
occurrence in the parentheses following them. 



General Positive Responses: 

• It would be an excellent example for beginning speech people or para-professionals. 

• I liked the examples that were natural. The example where the teacher sabotaged the 
situation by placing the sock on the hand instead of the foot was great! 

• Organization was excellent, following a clear and directed plan. 

• The product is very useful for learners. Parents and teachers who are being presented this 
information for the first time have a great deal to gain. Experienced individuals can use it 
as a review. 

• Camera work / lighting is good. 

• Very helpful for a university student in special ed, or in regular ed. 



Suggestions for Improving Audio-Visual Quality: 

• Video was fair for color, excellent for images, but cameraman made errors, like cutting the 
heads and bodies off and cutting in and out in views. 

• Viewers had different opinions about the pacing of the videos. Some thought it was slow 
(4) and boring sometimes, others thought it was a bit too fast especially for beginners. 

• Avoid narration on top of scenes that include audio. 

• Audio not consistent sometimes. (2) (The dubbing over of some things the teacher said 
was interfering. Sometimes no audio or even contradictory audio was present when 
teacher had mouth movements.) 

• Voice volume on audio low at times. 

• Text should be over the motion video with reduced luminance on video images. 
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Suggestions for Improving Content Quality: 

• Show more interesting examples. (5) 

• Longer video segments or slower pace should be provided for new concepts, or have an 
index with the current topic being discussed highlighted. (2) 

• Need the definition of the measure before going through the process. 

• Use subtitles. 

• Duration of the text on screen was too short on some sections. 

• Summarize all of the concepts at the end of the video for reinforcement. 

• Prerequisite knowledge for viewing makes it suitable for professionals rather than families. 

• Instructionally accurate but boring. 

• Use humor and do away with the monotone nature of the audio. 

• Model the behavior of the teacher on camera (more show not tell). 

• The written material should match the video exactly to improve clarity 

• Should do small group tiyout. (Follow one child through the steps to allow the learner to 
make sense of the process.) 

• Focus on a target audience, show the procedures being done by the “adults” rather than 
namation with random video clips, and talk to your learner instead of about them. 

• Socioeconomic level of students were not presented in an equal fashion. 

• Spend more video time on child interactions. 

• Closer attention to spelling errors in bulleted information. 

• A similar video with more examples of data keeping would be a nice follow-up. Also for 

middle school teachers on collecting data for specific behavior problems. 
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Suggestions for Improving Spanish Version: 

Needs more variety. 

Have a teacher narrate. 

Difficult to hear teachers and children. 

Make an all-Spanish version (text and nan-ation). 



Conclusions: 

We met our evaluation criteria by surveying several respondents in various categories (teclinical 
experts, teachers, parents) and asked them both rating-scale questions as well as open-ended 
questions relating to the appeal, effectiveness, and user-friendliness of the written, audio, and 
video materials in the SPIES video series. As in most evaluation studies, there was too great a 
variety of responses to represent every response here, but we have attempted to capture the most 
salient and frequently reported responses. 

On the topic of general quality and appeal, the video series rated “Good” with the majority of 
respondents. Overall, the technical experts rated the videos higher than did the learners, but the 
learners proffered more extensive comments and suggestions; we feel those, along with all the 
data we have gathered, will become a valuable asset in future production and 
development of videos like the SPIES series. 
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Video 1 



Mindy Anderson, Marla Bailey, and Cherie Doe 
EDUC 6550: Group Project Report, SPIES 

Dr. Nick Eastmond 
April 16, 1999 



The SPIES program was evaluated through the use of questionnaire and interviews. 
Interviewees included two learners and three experts. The following information includes the 
position, background, and the level of confidence of each of the interviewees: 





Position 


Learner 1 


Peer Tutor 


Learner 2 


Intervener; para- 
professional, 
student in Special 
Education 


Expert 1 


Special Education 
Teacher, severe 
unit 


Expert 2 


2"“' year graduate 
student in the 
Speech-Language 
Pathology program 
at USU 


Expert 3 


Communication 

Disorders 

Specialist 



O 




Backgroimd 


Confidence Level 


One year experience with 
High School age youth. 


Good 


One year experience with 
physically and 
intellectually disabled 
children ages 5-15 


Good 


Ten years experience as a 
Teacher, 2"^ year graduate 
student in the Speech- 
Language Pathology 
program at Utah State 
University 


Excellent 


Bachelor-of Science 
degree in Communication 
Disorders, 375 clinical 
clock hours, currently 
working in a severe unit 
preschool 


Good 


One and a half years 
experience with early 
intervention, preschool, 
one year experience with 
alternative kindergarten 
intervention, BS degree 
in Communication 
disorders, currently a 2"“' 
year graduate student in 
the Speech-Language 
Pathology program at 
USU 


Excellent 
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The results indicating the quality of the product are organized in the following table. The data 
was analyzed by averaging the number value (1-5, 1= very poor and 5= excellent) given by the 
interviewees. Questions from the original survey were assigned numbers (see master copy for 
number assignments) which were organized into the following categories: written, audio, and 
video. 





Written (Manual) 


Audio 


Video 


Appeal 


Learners: 4.5 avg 
Experts: 3.6 avg 


Learners: 4.3 avg 
Experts: 3.9 avg 


Learners: 4.5 avg 
Experts: 4.3 avg 


Effectiveness 


Learners: 4.5 avg 
Experts: 4.75 avg 


Learners: 4.25 avg 
Experts: 4.3 avg 


Learners: 4.4 avg 
Experts: 4.25 avg 


User-F riendliness 


Learners: 3.5 avg 
Experts: 4.75 avg 


Learners: 3.75 avg 
Experts: 3.6 avg 


Learners: 4.5 avg 
Experts: 4.0 avg 



Narrative Explanation 

• Generally, all interviewees rated from good to excellent, the written and video material 
as organized and adequately paced. 

• All interviewees rated from good to excellent, the effectiveness of the content being 
taught. It should be noted that the ratings of the written materials scored slightly higher 
than the video in effectiveness of material presented. 

• The instructional objectives were rated from good to excellent in regards to being clearly 
stated, however, the use of sufficient and relevant examples were rated from fair to good. 

Conclusions and Suggestions 

In conclusion, the following statements were given by the experts for further development. 

Additional comments are included: 

• “The dubbing over of some things the teacher said was interfering. If a certain phrase is 
wanted for use as an example, then it needs to be staged- otherwise, comment on the 
examples as they naturally occur.” 

• “Give more examples. You could also use poor examples or older techniques [that were 
developed prior to these newer techniques].” 

• “[The] duration of the text on screen was too short on some sections. Voice volume on 
audio was difficult to understand at times.” 

• “It was disconcerting at times to see the teacher moving her mouth, but no audio or even 
contradictory audio was present.” 
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• “For learners it would be helpful to summarize all of the concepts at the end of the video 
for reinforcement.” 

• I thought it was a good video tape. It would be an excellent example for beginning 
speech people or paraprofessionals.” 

• I liked the examples that were natural. The example where the teacher sabotaged the 
situation by placing the sock on the hand instead of the foot was great ! ” 

It was found that the experts generally gave lower ratings than those of the learners. This may 
indicate a target audience of parents or paraprofessionals would find the video useful. The 
experts tended to be more critical of the content than the target audience, however, the experts 
may want to use the material for training and inservice. 
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Video 1 (Spanish Version) 



Jennifer Merritt 
Wendy Warner 
Spies Project 
Spanish Edition 
April 17, 1999 



A. Data gathering process 

We asked two parents and two experts to watch the Spanish version of video 1 of the 
SPIES series, and answer the questionnaire. The parents were given the opportunity to 
ask questions, but the professionals viewed the videos alone on their own time. 

1. Experts 

a. Amy Burgoyne is an expert in the area of speech and language pathology. She 
works mostly with young children and is fluent in Spanish. 

b. Jen Merritt is an expert in the area of audiology. She works with children of 
all ages and is also a fluent Spanish-speaker. 

2. Learners 

a. Derek Jack is a new parent who speaks Spanish fluently. 

b. Ruth Jack is Derek’s wife. She also speaks Spanish. 

3. Level of confidence 

Both the experts and the learners appeared to be interested in answering the 
questions. The quality of responses was good. 



B. Results 





Audio 

Expert/Learner 


Video 

Expert/Learner 


Appeal 


• Narration uninteresting 

• Need Spanish audio 


• Use teachers more in teaching 
concepts 


Effectiveness 


• Subtitles are difficult to follow 


• Good content 

• Helpful ideas 


User- 

friendliness 


• All on-screen text and narration 
should be in Spanish 


• Who was the video made for? 

• Good translation 



No written materials were available in Spanish, so that column was not applicable. Also, 
because the experts and learners had very similar comments, we grouped the above results 
together. 

The main themes that we found were that the subtitles were hard to follow, and everyone 



agreed that the video should be dubbed in Spanish with Spanish on-screen text (not subtitles, but 
Spanish text in place of the English text). Another common comment was that the narration was 
a little boring. More teacher involvement was suggested. 

1 . Organization and Pace 

• Rated with 4s and 5 s 

• Spanish subtitles moved too quickly 

2. Effectiveness/objectives 

• Rated with 4s and 5s 

• Good content, but presentation was a little slow 

3. Other suggestions 

• More variety 

• All text and narration should be in Spanish 

C. Conclusions and Suggestions 

• Have a teacher narrate 

• Difficult to hear teachers and children 

• Make an all-Spanish version (text and narration) 

• Who was it made for 



Observations during interviews 



We were not present during the viewing by the experts 



learners 

• concentrated on video 

• stopped and asked questions 

• didn’t understand the purpose-after we discussed it they understood 

• made comments during the video 



Video 2 
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Formative Evaluations For Module 2, “Providing Help” 



“Providing Help” is designed for adults (teachers, day care providers, and parents) who 
work or live with preschool children who have disabilities, special health needs, or who are at 
risk for the development of a disability. In “Providing Help” four forms of help are defined and 
illustrated using examples of adults interacting with preschool children in natural settings 
(home, school, or stores). Three help strategies are also defined and illustrated using examples 
of adults interacting with preschool children in natural settings. 

Data Gathering Process 

Three learners and three experts in the fields of early childhood education and child 
development were interviewed. The interview consisted of viewing the video with each person, 
asking the participants for their comments, and having the participants complete a questionnaire. 



A. Interviewees - Learners 

1. Karmen Blanthom is an Elementary Education Teacher with a Special Education 
Endorsement. Karmen is very knowledgeable about issues in special education. I am very 
confident with her evaluation. 

2. Tifiiee Anderson is a parent and a preschool teacher. Tifhee has worked with autistic 
children. I am confident with her evaluation. 

3. Debbie Pugsley is a parent and a Special Education Aide. Debbie works with autistic children 
and students with special needs. I am confident with her evaluation, but she is still new in the 
field and is very much a learner. 

B. Results 



Quality of the Product 





Written 


Audio 


Video 


Appeal 


• Nice outlines 

• Good supplement 


• Sufficient 


• Good sequencing 

• Is it about over? 


Effectiveness 


• Readable 

• Comprehensive 


• Audible 

• Dry at times 


• Common sense 
information 

• Real life examples 


User-Friendliness 


• Easy-to-follow 


• No coirunents 


• Easy-to-follow 

• Good outlines for 
beginners 



Written- The information was very complete, providing a good outline to be followed during 
possible follow-up learning sessions. The material could act as a supplement for other programs 
currently being used. It was readable and easy-to-follow. 

Audio- All comments centered on the fact that it could be heard. The background music used 
was not the most liked at times. 

Video- The video was very well organized, following clear objectives. Most of the information 
presented was common sense, and left the viewers wondering how long this was going to 
continue. Are we going to learn anything new? 

7. The pacing was slow at times. Viewers felt it was dragging and needed to end or move on. 
Or ganizati on was excellent, following a clear and directed plan. 

8. The product is very useful for learners. Parents and teachers who are being presented this 
information for the first time have a great deal to gain. Experienced individuals can use it as a 
review. 

9. Possible adaptation of the skills presented to older students and extension of skills to regular 
education students. Excite the narrator! I Use humor and do away with the monotone nature of 
the audio. 

C. Conclusions and suggestions 

• Pace was too slow 

• Excellent organization 

• Product is very useful 



A. Interviewees - Experts 

1 . Barb Fiechtl is an instructor in the Department of Special Education at Utah State University. 
Barb has a Masters Degree in Special Education, and has been involved in the field for 24 years. 
Her expertise is in the areas of early childhood education and the education of children with 
severe needs. For these reasons, I have a high degree of confidence in Barb’s responses. 

2. Shelley Lindauer is an Associate Professor of Family and Human Development and the 
Director of the Child Development Lab at Utah State University. Shelley has a Bachelors degree 
in Early Childhood Education and both her Masters and Ph.D. are in Child Development. She 
has worked at Utah State for 16 years and has worked in the field for 24 years. I have a high 
degree of confidence in Shelley’s responses. 

3. Marti Dever is an Associate Professor in the Department of Elementary Education at Utah 
State University. Marti has a Masters in Early Childhood Education and an Ed.D. in Education. 
She has worked at Utah State for 6 years and has worked in the field for 29 years. I have a high 
degree of confidence in Marti’ s responses. 



B. Results 



Quality of the Product 





Written 


Audio 


Video 


Appeal 


• Easy-to-follow 

• Understandable 


• Great 

• Pleasant voice 


• Very appealing 


Effectiveness 


• Written clearly 

• Helpful 


• Great 


• Excellent 
° Great material 


User-Friendliness 


• Very user friendly 

• Jargon-free 
language 


• Understandable 


• Good examples 

• Paced well 



Written- The experts thought the manual was easy-to-follow, understandable, and helpful. They 
felt that the material was presented in simple, jargon-free language. Two out of three thought 
the use of advanced organizers was excellent and everyone thought the use of objectives was 
excellent. 

Audio- There were very few comments made specifically in regard to the audio portii n. One 
person commented that the audio was very appealing and the narrator’s voice was pleasant. 

Video- Two out of three experts rated the video as being excellent in presenting up-to-date 
information. All three commented on the use of good examples throughout and thought that the 
material was paced well. One person especially liked the idea of using natural settings. 

7. The experts thought that the material was well-organized and was paced very well. 

8. One expert said that she thought the program taught the subject matter very well. Two of the 
three experts said that they thought a lot depended upon the facilitator. One person felt some 
clarity was needed in regard to assessing students’ needs for help. Everyone agreed that the 
objectives were clearly stated. One person thought that more examples would be helpful to 
illustrate verbal help. 

9. Some suggestions experts had for future development were to provide guidelines for using the 
help strategies, such as, “This strategy is good when. . .” Also, to focus more questions on such 
as. When? Why? What to do? How? One person suggested to replace the word “age” with 
“stage”in the Introduction to SPIES, under “Naturalistic Methods of Instruction”, where it reads, 
“. . . Developmentally appropriate activities are those that other children of the same age choose 
to do and that promote learning and development by keeping children actively engaged.” 

Finally, one person suggested interspersing more video examples with the graphics so viewers 
are not just looking at words on the screen. 
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C. Conclusions and suggestions 

• Well-organized and paced well 

• Objectives are clearly stated 

• Provide guidelines for using help strategies 

Interview Notes 

1. Before Barb Fiechtl and I began watching the video, we agreed to pause after section so she 
could share her responses. Here are some of Barb’s comments: 

The examples provided in the section Physical Help were good, except for two. In the first 
example a parent was demonstrating the use of full physical help by using hand-over-hand 
guidance. In this example the visual is not effective because you can’t see what the parent’s 
hands are doing. In the second example a teacher was demonstrating the use of partial physical 
help. The teacher is shown helping a student pour juice, but the teacher appears to be providing 
full physical help instead of partial physical help. 

The television we used had closed captions. Barb mentioned that the “g’s” and “y’s” were 
hard to read in the captions. We noticed one example in which the closed caption blocked the 
view of the intervention that was being demonstrated. 

The examples in the section Verbal Help could be improved. For instance, vary the examples 
by giving children opportunities to complete tasks other than verbal tasks. Also, provide some 
isolated examples of direct verbal help and indirect verbal help. 

In the section Nonverbal Help, Barb’s suggestion was to give children opportunities to 
respond without prompts. 

In the Most-To-Least Help Strategy section, the authors state that rationale for this strategy is 
to encourage children to respond with few or no errors. Barb disagreed with this statement, she 
said that the Most-To-Least strategy is to designed to produce errorless learning. 

In the Least-To-Most section Barb commented that some of the scenes were dark. Also, the 
mother in the examples speaks to the child, but doesn’t use prompts. 

In the final section on Progressive Time Delay, Barb felt that the examples were good. 
However, they don’t provide the viewers with any idea of how long this process takes. 

Finally, Barb thought the video could be strengthened by providing the viewers with 
guidelines for using the various strategies, such as, “This strategy is good when. . .” 
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2. Before Marti Dever and I began viewing the film, we agreed to pause between sections so she 
could share her responses. Here are some of Marti’s comments: 

Marti liked the notion of doing things in a natural setting. She said she thought that it was 
very appropriate. Marti thought that the examples in Physical Help seemed logical. Also, she 
thought that the examples presented a nice way to think about scaffolding children’s learning. 

Marti commented that the distinction between direct and indirect help in Verbal Help seemed 

fuzzy. 

Marti felt that some discussion on when to use Non-Verbal Help would be helpful. Also, she 
wondered how this continuum worked. She thought that there should be some mention of doing 
an initial assessment on the child to determine where they are. 

Marti thought the examples in the Least-To-Most Help Strategy section were more clear than 
the examples in the Most-To-Least section. She commented that it would be helpfiil to have 
some guidelines on which strategy to use. 

Marti’s question after watching the Progressive Time Delay Strategy, was, ‘ Is precision that 
critical?” 

In response to the question, “How well does the program teach?”, Marti replied that a lot 
depends on the facilitator. She liked the material. She thought that the objectives were stated 
clearly. 

The only suggestion Marti had was to change the word “age” to “stage”, as I described above. 

3. Shelley Lindauer and I agreed to pause the video after each section so she could share her 
responses. Here are some of Shelley’s comments: 

In the Physical Help section, Shelley thought the examples were good with enough variety to 
keep it interesting. She liked that some examples were in school and some examples were in the 
home. She also thought that the examples provided good modeling. 

Shelley’s question after viewing the section on Modeled Help, was, “How does a parent or 

teacher know when to give full or partial help?” 

Shelly thought it would be helpful to have more examples in the Verbal Help section. She 
said that this is a difficult thing to teach teachers because so much is intuitive. On the other 
hand, Shelley thought the examples provided in the Non-Verbal Help section were good. 
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In the Most-To-Least Help Strategy section, Shelley thought the examples were effective and 
that is was a great sequence using the same child. Similarly, Shelley thought that the examples 
were really good and paced well in the Least-To-Most section. 

Shelley thought that it was a neat idea to show a sibling in the examples for Progressive Time 
Delay. However, she didn’t feel that the examples were as strong as the other two help 
strategies and that the example seemed more “set up”. 

Shelley thought that the overall quality was very good, very user-friendly, clear, and moved 
quickly. Her suggestion for future development was to provide suggestions for determining why 
you do what you do and when you do it. 
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Strategies for Preschool Intervention 
Video Expert and Learner Review 

lA. Describe the data gathering process you used 



Module 3: Incidental Teaching 
Judy A. Ricks 



Data was gathered from two-video experts and two-learners. The video experts direct, 
produce, and edit training videos for adult learners. Additionally they are instructional designers 
and have developed paper-based training. One of the video experts is a graduate of the 
Instructional Technology Department at Utah State University. The other expert is a graduate of 
Utah State Teacher Education program. Each has between 12 and 8 years in instructional design. 
They understand the concepts of learning theory, test and measurements, and objectives. The 
video experts selected have been in the training and video field for 12 and 8 years respectively. 
Because the individuals were expert in the training video business and have developed numerous 
training videos for instructional purposes I felt there input would be worthwhile and they could 
give honest and critical feedback. On a scale of 1 to 5 my confidence in their ability to rate the 

video production and visual components a 5. 

The two learners are both parents with two children each. They are also instructional 
designers but they do not design in the area of small children and their children are no longer of 
preschool age. However the two learners do understand objectives, testing and measuring, and 
the basic laws of learning because of their work in instructional design. On a scale of 1 to 5 my 
confidence in their ability to rate the material objectively and accurately is a 3. I rate them a 3 
because of their lengthy work in adult education. After the interview I believe their extensive 
work in adult education corrupted their judgement to fairly evaluate a video concerning 
preschool intervention. Additionally their children are beyond the preschool years. The biggest 
factor in rating my confidence level a 3 is all their work in adult education. I think they find it 
hard to separate the learning concepts and rightfully so. But overall the evaluations performed 
were an honest attempt to rate the training video fairly and accurately. 




Page 1 of h 



28 



Strategies for Preschool Intervention 
Video Expert and Learner Review 

IB. Results from the video (technical) experts. 



Module 3; Incidental Teaching 
Judy A. Ricks 





Written Material 


Audio 


Video 


Aoneal 


Rated Good Overall 


Rated Fair Overall 


Rated Good Overall 


Effectiveness 


Rated Fair Overall 


Rated Good Overall 


Rated Good Overall 


User Friendlv 


Not Rated 


Rated Good Overall 


Rated Fair Overall 



Figure 1. Results from the Video Experts 
Data Gathered from the two-video experts reiterated below in bulleted format 



Production Components 

^ Quality of images, 3 blocks marked as fair, 3 blocks marked as good 
Audio, 1 block marked as fair, 4 blocks marked as good 
Up-to-date information, rated as good, all 4’s, overall 



Visual Components 

Transparent camerawork 1 block marked as poor and 1 block marked as good. 

Lighting rated as good, all 4’s, overall 

On screen readability rated as good, all 4’s, overall 

Overall quality of visual images rated as fair, all 3 ’s, overall. 



• Instructional Content 

Use of advanced organizers not rated and rated as good 

Use of objectives, sticking to the main topic, adequate time for mental processing, 
use of understandable language rated as good, all 4’s, overall. . 

Use of sufficient relevant examples rated as fair, all 3’ s, overall. 

• Affective Components 

Presentation of Material rated as poor, all 2’s, overall 

Material holds the interest throughout rated as poor, all 2’s, overall 

Applicability rated as fair, all 3’s, overall 

Humor not rated in either evaluation or marked as N/A, not applicable 
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Strategies for Preschool Intervention Module 3: Incidental Teaming 

Video Expeii and Learner Review •'“"V 



IB. Results from the learners. 





Written Material 


Audio 


Video 


Appeal 


Rated Good Overall 


Rated Fair Overall 


Rated Good Overall 




Effectiveness 


Rated Fair Ov^erall 


Rated Good Overall 


Rated Good Overall 


User Friencflv 


Rated Good Overall 


Rated Good Overall 


Rated Fair Overall 



Figure 2. Results from the Learners 



Production Components 

Quality of images rated as good, all 4’s, overall 

Audio, 1 block marked as poor, 2 blocks marked as fair, 3 blocks marked as good 
Up-to-date information, 1 block marked as fair and 1 block marked as good 



Visual Components 

Transparent camerawork, rated as good, all 4’s, overall 
v' Lighting rated as good, all 4’s, overall 

On screen readability 1 block marked as fair and 1 block marked as good 
^ Overall quality of visual images 1 block marked as fair and 1 block marked as good 



Instructional Content 

^ Use of advanced organizers 1 block marked as fair and 1 block marked as good 

Use of objectives, sticking to the main topic, adequate time for mental processing, 
use of understandable language rated as fair, all 3’s, overall with one block, use of 
objectives rated a poor, a 2 

Use of sufficient relevant examples rated as fair, all 3’s, overall. 




Affective Components 

^ Presentation of Material, as very poor and poor 

Material holds the interest rated as very poor and fair 
Applicability rated as poor and good 

Use of appropriate humor rated as fair and a question. Is their humor in PHD work? 

Page 3 of 8 

30 



Strategies for Preschool Intervention 
Video Expert and Learner Review 



Module 3: Incidental Teaching 
Judy A. Ricks 



Conclusions and suggestions 
Comment: Visually the work was slow and boring 

Comment: Technically it worked all its objectives 

Comment: Camera work/lighting was pretty good. 

Comment: Show some interesting examples to reinforce the subject we have already been 
introduced to. 

Comment: Instructionally accurate but boring. 
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Sfrategies for Preschool Intervention 
Expert Review 



Module 3: Incidental Teaching 
Dixie Jenson 



SPIES; Module 3 Evaluation - Incidental Teaching 
Description of data gathering process - expert interview 

This segment of the Module 3 evaluation was taken from interviews with two experts in 
the field of Special Education. Both individuals have a Master’s in Early Childhood Special 
Education and a M. Ed. They have between 8 and 17 year’s background teaching children who 
have special needs. Both have had the opportunity to evaluate educational materials to be used to 
train para-professionals and parents. I feel both of the expert evaluator’s were well qualified to 
give professional evaluations. 

Questions were determined prior to the interviews to establish continuity in the evaluation. 
One interview took place in a confidential ofiBce area and the other one in a home. The interview 
questions where focused on three main components of the Module 3 : production components, 
visual components and the written materials. An additional area was provided for the individuals 
to write any suggestions or comments. 

Results of evaluation 



Quality of the Product 



Ratings: l=verypoor 2-poor 3=fair 4=good 5=excellent 






Written Manual 


Audio 


Video 


Appeal 


3, 4 


4,3 


3,3 


Effectiveness 


3,4 


4,3 


4,3 



Strategies for Preschool Intervention 
Expert Review 



Module 3; Incidental Teaching 
Dixie Jenson 











U ser-fiiendliness 


3,3 


n/a 


n/a 



Narrative explanation of bulleted points 

Written materials: Both experts rated the written materials in this area between fair and 
good. Some of the written comments were: some terminology may be unknown (e g. stereotype), 
more examples beyond reiterating the video examples would be helpful, written materials were 

both evaluated as well outlined and informative. 

Both evaluators were concerned with the professional language that would be difficult for 

parents to follow. 

Audio components: Again the evaluators rated this area between fair to good. If we 
include up-to-date language we find a fair to good rating. Both evaluators commented about 
difficulty in hearing child on step with mother. They could not hear the children s responses. 
Audio portion was not as effective as the video portion. Music did not synchronize with the 
beginning of some text. 

Video: The production quality of the video received a good rating. But quality of visual 
images received a fair to good rating. Both evaluators commented on the difficulty seeing the 
quality of video with the child reaching in the circled out portion. They both felt this was 
confusing to the viewers. It may have been better to leave out the circle around the child. Text 
was clear on the screen but there was confusion because audio voice was moving at too rapid a 
pace. 

iO 
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Strategies for Preschool Intervention 
Expert Review 

How well is the material organized and paced? 



Module 3; Incidental Teaching 
Dixie Jenson 



^ Text was clear on the screen but there was confusion because audio voice was moving at too 
rapid a pace. 

Music did not synchronize with words when they first appeared on the screen on a few 
occasions. 

Manual was well organized but should be pre-read or followed closely during video 
presentation if participants are non-professionals. 

Pacing was a little to fast and viewers needed to review sections to see if they understood the 
concepts. 

^ Both thought the material was in the fair range for the way the material was organized and 
ease to follow. 

How well does the product teach? 

Both evaluators felt that a person would have to have an educational background in the field 
and be very familiar with the terminology to keep pace with the video and understand what it 
was teaching. 

^ Instructional objectives had a good rating for being clearly stated. 

Time was allocated for discussion time during video presentation for audience to ask 
questions. 

Suggestions and Comments for modifications and future development - expert evaluators 

It was confusing to mix other children’s examples in with Ellie’s hand washing scene. It 

would be better to keep her instruction separate from the other’s and just use her at the 
beginning as an example of the proper way to do incidental teaching. 

It would be good to avoid narration on top of scenes that include audio - Ellie’s scene. 
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Strategies for Preschool Intervention Module 3: Incidental Teaching 

Expert Review Dixie Jenson 

It would be better to get better quality audio recording of children’s responses so that they 



could be heard on the video. If they can’t be heard, then maybe they should use subtitles. 

S Note: Both evaluators were told and could see from the title on the written material and the 
video portion of the instruction that the SPIES materials were designed for families and 
professionals working with children with special needs. At the conclusion of the viewing and 
reading the materials both left the following explanations: 

Comment: The person viewing this material would have to have an educational background 
and be very familiar with the terminology to keep pace with the video. The video was geared 
more toward professionals and not user friendly to families. 

Comment: If the program is for parent use, there needs to be more explanation, slower pace, 
slower talking. It’s for parents to follow and get help. 

Comment: Longer video segments to follow the concepts - especially for parents. 

^ Comment: Avoid narration on top of scenes that include audio (Elbe’s scene). 

Comment: If child verbal initiation or responses are hard to understand/hear, use subtitles. 



Education 6550 - Dr. Eastmond 
April 14, 1999 
Dixie Jenson 

SPIES: Module 3 Evaluation - Incidental Teaching 
Description of data gathering process - expert interview 

This segment of the Module 3 evaluation was taken from interviews with two experts in 
the field of Special Education. Both individuals have a Master’s in Early Childhood Special 
Education and an M. Ed. They have between 8-17 years background teaching children who 
have special needs. Both have had the opportunity to evaluate educational materials to be used to 
train para professionals and parents. Is feel both of the expert evaluator’s were well qualified to 
give professional evaluations. 

Questions were determined prior to the interviews to establish continuity in the 
evaluation. One interview took place in a confidential office area and the other one in a home. 
The interview questions where focused on three main components of the Module 3: production 
components, visual components and the written materials. An additional area was provided for 
the individuals to write any suggestions or comments. 

Results of evaluation 



Quality of the Product 



Ratings: l=very poor 2-poor 3=fair 4-good 5-excellent 






Written Manual 


Audio 


Video 


Appeal 


3, 4 


4,3 


3,3 


Effectiveness 


3,4 


4,3 


4,3 


U ser-friendliness 


3,3 


n/a 


n/a 
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Narrative explanation of bulleted points 

Written materials - both experts rated the written materials in this area between fair and 
good. Some of the written comments were; some terminology may be unknown (e.g. 
stereotype), more examples beyond reiterating the video examples would be helpful, written 
materials were both evaluated as well outlined and informative. 

Both evaluators were concerned with the professional language that would be difficult for 
parents to follow. 

Audio components - again the evaluators rated this area between fair to good. If we 
include up-to-date language we find a fair to good rating. Both evaluators commented about 
difficulty in hearing child on step with mother. They could not hear the children’s responses. 
Audio portion was not as effective as the video portion. Music did not synchronize with the 
beginning of some text. 

Video - the production quality of the video received a good rating. But quality of visual 
images received a fair to good rating. Both evaluators commented on the difficulty seeing the 
quality of video with the child reaching in the circled out portion. They both felt this was 
confusing to the viewers. It may have been better to leave out the circle around the child. Text 
was clear on the screen but there was confusion because audio voice was moving at too rapid a 
pace. 

How well is the material organized and paced? 

- Text was clear on the screen but there was confusion because audio voice was moving at too 
rapid a pace. 

- Music did not synchronize with words when they first appeared on the screen on a few 
occasions. 




- Manual was well organized but should be pre-read or followed closely during video 

presentation if participants are non-professionals. 

- Pacing was a little to fast and viewers needed to review sections to see if they understood the 

concepts. 

- Both thought the material was in the fair range for the way the material was organized and ease 

to follow. 

How well does the product teach? 

- Both evaluators felt that a person would have to have an educational background in the field 

and be very familiar with the terminology to keep pace with the video and understand 
what it was teaching. 

- Instructional objectives had a good rating for being clearly stated. 

- Time was allocated for discussion time during video presentation for audience to ask questions. 



Suggestions and Comments for modifications and future development - expert evaluators 
- It was confusing to mix other children’s examples in with Elbe’s hand washing scene. It would 
be better to keep her instruction separate from the other’s and just use her at the 
beginning as an example of the proper way to do incidental teaching. 

-It would be good to avoid narration on top of scenes that include audio - Elbe’s scene. 

-It would be better to get better quality audio recording of children’s responses so that they could 
be heard on the video. If they can’t be heard, then maybe they should use subtitles. 

Note; Both evaluators were told and could see from the title on the written material and the 
video portion of the instruction that the SPIES materials were designed for families and 
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professionals working with children with special needs. At the conclusion of the viewing and 
reading the materials both left the following explanations: 

Comment; The person viewing this material would have to have an educational 
background and be very familiar with the terminology to keep pace with the video. The video 
was geared more toward professionals and not user fnendly to families. 

Comment; If the program is for parent use, there needs to be more explanation, slower 
pace, slower talking. It’s for parents to follow and get help. 

Comment: Longer video segments to follow the concepts - especially for parents. 

Comment: Avoid narration on top of scenes that include audio (Elbe’s scene). 

Comment: If child verbal initiation or responses are hard to understand/hear, use subtitles. 



Video 4 
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Cloe Russell 
Lory Farr 



Evaluation of Module 4 SPIES Video: 
Tracking Progress 



I. Evaluation questions and data gathering process. 

A. Persons Interviewed 

1. Expert 

• Currently directs the filming of educational videos at USU’s CPD 
department.. Has a B.A. in Film production and M.S. in Instructional 
Technology. Has had 9 years experience in video/film and 3 years in 
Instructional Development. 

2. Teachers 

• Teacher #1 - Dr. of Education who has had experience with all ages for the 
past 40 years. He is well-renowned in behavioral science, child care, and 
authored many books and articles on this subject. Has had experience in 
making videos of this type. 

• Teacher #2 - B.S. in Family and Human Development/Psychology. Three 
years experience with adults with behavior problems as director of 
program. Two years working on similar project. Teacher of behavior 
management to paraprofessional. 

• Teacher #3 - Middle school special education teacher. Has a B.S. in 
Special Education. Has taught for 14 years in public schools. 

3. Parents 

• Parent #1 - Special Education Aid for five years at a public school district, 
responsible for main streaming adolescents 12 years of age and up from 
education settings to employment. 

• Parent #2 - Parent of a two and a five-year old. Has had experience as a 
para professional in a special education classroom. Has three years of 
college and majored in Communicative Disorders. Presently works in the 
home. 
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B. Results 

1. Written materials 

• Teacher #1 mentioned that the written material should be more consistent 
with the video material (see post-it notes). 

2. Audio 

• Five interviewees gave the quality of audio a “4" and one gave it a “5.” 

3. Video 

• Our expert gave a “2" for transparent camera work, “5" for sufficient 
lighting, and “4" for readability and image quality. 

• Teacher #1 had no comments about the quality of the video in addition to 
his rating of “4” on the scale he circled. 

• Teacher #2 gave a “5" for lighting and visual images. She said that she 
saw it on a large screen TV, and on a small screen, it may be too small. 

• Teacher #3 gave a “4" for lighting and a “5" for readability and visual 
images. 

• Parent #1 mentioned that the video was not focused in certain places and 
that it might help to continually show a written index on the screen to give 
a better sense of orientation. 

• Parent #2 gave all “4's” on the visual. 

C. Conclusions and Suggestions 

• Our expert commented on effectiveness and suggested that the text be 
over the motion video wUh reduced luminance on video images. Also 
model the behavior of the teacher on camera (more show not tell). The 
quality of the video was fair for color, excellent for images, but felt that 
the cameraman made errors, like cutting the heads and bodies off and 
cutting in and out in views. 

• Teacher #1 thought the content was good but had the following comments: 
The written material should match the video exactly to improve clarity. 

The pacing might be too fast for beginners. The video duration should not 
exceed twenty minutes. 

• Teacher #2 suggested that to make it user friendly, be less “Techie.” 

Follow one child through the steps to allow the learner to make sense of 
the process. Steps would seem more relevant. Video needs to show what 
narrator is saying. There is too much information packed into one 
sentence. If s intimidating. In summary, slow down, follow the procedure 
through with one person, focus on a target audience, show the procedures 
being done by the “adults” rather than narration with random video clips, 
and talk to your learner instead of about them. 

• Teacher #3 though the video would be very helpful for a university student 
in special ed, or in regular ed. A similar video with more examples of data 
keeping would be a nice follow-up. Also for middle school teachers on 
collecting data for specific behavior problems. 

• Parent #1 would like more examples, to go a little slower when 
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introducing concepts and that it would help if on the video it was clearly 
indicated where they were in the written material i.e. “...on page 23 of the 
manual...” or to have an index somewhere on the screen with whatever 
topic being discussed highlighted in some way. 

Parent #2 said it was difficult to follow at the beginning. Need the 
definition of the measure before going through the process. (She had to 
make an outline to follow the video sequence.) As a parent only I would 
not understand why any of this is important, but her background helped 
her. The examples were not necessarily appropriate for age. 



,1 



Video 5 







44 



SPIE^ Project 



A. 

Position 


Learner #1 

parents 


Learner #2 

parents 


Expert #3 

grade teacher 


Expert #4 

Preschool teacher 


Background 


parent of 2 
BS ComD 


parent of 4 


grade teacher 
28 years teaching 


1st year teacher 


Confidence 


high 


high 


high 


medium 


B. 













Written Manual 


Audio 


Video 


Appeal 


It wasn't available to our 
viewers at the time of the 
evaluation. 


Quality of Audio GOOD 


Sharpness, lighting, color 
quality, and content 8ill 
judged to be GOOD TO 
EXCELLENT 


Effectiveness 




GOOD 


GOOD 


User Friendliness 




GOOD 


FAIR TO GOOD 



WRITTEN MANUAL 

• We were unaware that our viewers needed to have access to the written material as they watched the 
video. With several of our viewers out the area we were unable to have them reevaluate the video with the 
written material. 

AUDIO 

• All of the viewers rated quality of audio as good. One viewer commented that the narrators voice sounded 
“slushy.” One viewer commented the rate was a little fast. 

• All of the viewers indicated that they gedned information from the video. 

• All the viewers indicated the objectives were clearly stated, and the material was understandable. 

VIDEO 

• Most of the ratings on transparent camera work, sufficient lighting, readability of onscreen text, and 
quality of visual images received ratings of good and excellent. 

• The organizers, objectives, and applicability of information were edl rated as good. The use of examples in 
the video appeared to be helpful, however, one viewer commented that the “goals of early intervention” 
needed more examples. 

• The language was understandable, up-to-date, and appeared to be presented in an interesting manner. 

7. The material was judged to be organized in an excellent manner. One viewer commented the rate was 
a little fast. 

8. The product teaches in an appropriate manner. The objectives were clearly stated and addressed 
throughout the video. The material was judged to be applicable to the viewers. More examples of the 
techniques presented would have been helpful during the video. 

9. Presentation of further examples would be helpful in the future. 



Overall the video appeared to be presented in a sufficient and informative way. It was user friendly and the 
audio and video components were judged to be good overall. Suggestions included commenting on the 
importance of home involvement with these children. One viewer also indicated that the socioeconomic level 
of these students were not presented in a equal fashion. For example, the “white children appeared to have a 
higher socioeconomic level and have less medical challenges visually.” More examples throughout the video 
would also be helpful in future productions. 
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Video 6 
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Dan Trimble / Jeramy Jenkins 
Research for the Classroom Teacher 
Project: SPIES 



Video Evaluation: SPIES 



Data Collection Process: 

Position of Experts : 

1. Audiophile, vwth emphasis in High Fidelity audio. Has no formal training, but 

is being trained as an audiologist. 

2. Graduate student in Audiology. Relevant expertise includes training in 

principles of audio-science, acoustics, and various audio instrumentation. 

• Level of Confidence: Neither of these individuals are audio or video engineers as 
such but confidence in their opinions, given their experience, is high. 

Learner Background and Experience : 

1. Certified teachers assistant. Relevant training and history include an 

Associate degree in education with emphasis in elementary age 
populations. 

2. Bachelors in Family and Human Development with a minor in special 

education. 

• Level of Confidence: Both of these individuals have training and experience in the 
areas addressed by the video. We have great confidence in the accuracy and 
appropriateness of their responses. 



Results: 





Written 


Audio 


Video 


Appeal 


1. Good 


4. Fair 


7. Fair/good 


Effectiveness 


2. Good 


5. Fair 


8* Good 


User- 

Friendliness 


3. Good 


6. Fair 


9. Good 



Narrative Explanation: 

1. The data from this category was derived from 3 questions. The mean of the scores for 

clarity was 3.75 on a scale of 1-5 with five being excellent. The mean of the scores for 
helpfulness was 4.5 or good. The mean of the scores for the appeal to users was 4.25 
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or good. 

2. The mean of the scores for helpfulness was 4.5 or good. 

3. The mean of the scores for clarity was 3.75 on a scale of 1-5 with five being exceUent. 
The mean of the scores for helpfulness was 4.5 or good. The mean of these scores 
being 4. 1 which is good. 

4. The score for this category was derived from the audio quality question on the 
questionnaire and from numerous comments. The mean score for the questionnaire 
was 3.75 or good. The comments regarding the music referred to it’s repetitive nature 
which was reported as distracting from the content. The mean score for the comments 
regarding the music was 2.3 or poor. 

5. The score for this category was derived from the audio quality question on the 
questionnaire and from numerous comments. The mean score for the questionnaire 
was 3.75 or good. The comments regarding the music referred to it’s repetitive nature 
which was reported as distracting from the content. The mean score for the comments 
regarding the music was 2.3 or poor. 

6. The score for this category was derived from the audio quality question on the 
questionnaire and from numerous comments. The mean score for the questionnaire 
was 3.75 or good. The comments regarding the music referred to it’s repetitive nature 
which was reported as distracting from the content. The mean score for the comments 
regarding the music was 2.3 or poor. 

7. These scores were derived from the questions in the questionnaire relating to video 
appeal. The mean of the scores for this category were 3.69. 

8. These scores were derived from the questions in the questionnaire relating to video 
effectiveness. The mean of the scores for this category were 3.8. 

9. These scores were derived from the questions in the questionnaire relating to video 
user-friendliness. The mean score was 4.25. 

Conclusions and Suggestions: 

Based on unanimous responses, planning for the next video series should include more 

engaging music and visual fields backing the bulleted information sections. Other suggestions 

include: 

1 . Spending more video time on child interactions. 

2. Closer attention to spelling errors in bulleted information, etc. 
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Appendix B: Sample Questionnaire 



VIDEO EVALUATION FORM FOR SPIES VIDEOS 

Video Title: 

Proposed Audience: 

Length of Video: 

Video’s Objectives: 

Description of Video Content: 



Please rate the following elements of the video on a scale of 1-5 

l=very poor 2=poor 3=fair 4=good 5=excellent 



Production Components: 

Color quality of images 

Images in focus/Sharpness of images 

Quality of audio 

Up-to-date information 

Visual Components: 

Transparent camera work 
Sufficient lighting 
Readability of on-screen text 
Quality of visual images 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



Instructional Content: 



Use of advanced organizers 
Use of objectives 

Cohesive contentA^ideo sticks to main topic 
Adequate time for mental processing of information 
Use of understandable and up-to-date language 
Use of sufficient and relevant examples 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



Affective Components: 

Presentation of material is interesting 
Presentation of material holds interest tliroughout 
Relatability/applicability of information 
Use of appropriate humor 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 




EXPERT CHECKLIST FOR SURVEY RESPONDENTS 



Please circle one: 

Respondent Group= Parent Teacher 

If Technical Expert, please indicate your area of expertise: 



Technical Expert 



Proposal for Formative Evaluation of the SPIES Program 

Purpose and Objectives: 

The purposes for the evaluation of the SPIES program are to answer the following 
evaluation questions: 

1. What is the quality of the product: 

. The written material 

• The audio 

• The video 
with the criteria of: 

. appeal to user 

• effectiveness 

• user-friendliness 

7. How well is the material organized and paced? 

8. How well does the product teach? Are instructional objectives clearly stated and 
followed up with help to meet them? 

9. Other questions like suggestions for future development which will be asked to 
experts only. 

Steps / Procedures: 

The class is to evaluate the whole SPIES series by conducting expert interviews and one- 
on-one trial with learners. 6 groups of 2 students will each take one video out of the 
SPIES product and evaluate them with the following steps: 

1 . Familiarize themselves with the materials as well as possible; 

2. Select 2-3 experts and to interview with Questionnaire attached; 

3. Select 2 — 3 learners for one-on-one-trial with Questionnaire attached; 

4. Report data and results of findings 
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